Papers with sparse model
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2022.findings-emnlp)
Copied to clipboard
Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, Wen-tau Yih
| Challenge: | Existing sparse retrievers lack the ability to match salient phrases and rare entities in the query. |
| Approach: | They introduce a dense Lexical Model that can be trained to imitate a sparse one. |
| Outcome: | The proposed model outperforms sparse retrievers on a range of tasks including five question answering datasets and the MS MARCO passage retrieval. |
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing tasks. |
| Approach: | They propose a parameter-efficient sparsity crafting method which crafts dense models into sparse models using the mixture-of-experts architecture. |
| Outcome: | The proposed method significantly reduces computational costs and GPU memory requirements, while maintaining the quality of approximation in function space. |
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a reward model (RM) to improve the effectiveness of large language models, there are two challenges in training. |
| Approach: | They propose a reward model (RM) that is a proxy of human preferences and assigns scores to the outputs of the large language model (LLM) a human annotation consistency rate of 60% to 75% is causing training data to contain a lot of noise. |
| Outcome: | The proposed model outperforms state-of-the-art ensemble methods and mitigates the overoptimization problem. |